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HISTORICAL USES FOR STUDENT .EVALUATION 

V Tlie notion that it might in some way be useful, ar at least interesting, 
to seek out information from students concerning what they think about the 
current instructional experiences in which th^y are involved is far from new. 
Centuries ago susii data, was not only sought out. but was in some inst-ancos 
used for purposes which involved the levying of fines upon faculty members ' 
(Rashdall. .1936). At Americ|ji universities, while the results were not as 
-seveFer-^WenrraOT^^^ o^r tl^ach-eTs- were-b-etng e^^^^^^^ 



ago (Centra, .1972). Rather extensive hist^ies^of the student evaluation ~ 
movement have been compiled elsewhere (Werdell. 1967. Ebel. 1976).. the 
-purposes for gathering student evaluations have ranged f^om pure intellectual 
curiousity to the aforementioned seeking of evidence to be used in meting- 
out subsequent punishment to^ilatory .instructors. For the most parfhowever . 
categories of purposes in accumulating student ratings of instruction can 
be reduc-e-d-to three. They involve "1) 'provision of feedback to faculty.with a 
•vi-ew toward- modification and improvement ok instructioi;, 2). acquistion of 
information to be -used t^^.evaluate faculty -in some manner and to make decisions 
wlLU i.'egatJ.U . tenure, p.^m'Ltion .gnd j^i nfi j^^^ concerns, and 3) dis- _ 
simination to students of. data thought ^to be helpful as they make choices 
viz a viz various course offnrings and curricufum options (McK^achie. et al.. 1975). 



It has been pointed out elsewhere (Ronan, 1973)' that students are cer- 
tainly in a position to observe at close range a large quantity of teaching 
arid are in a good position to provide some reactions to what they exper3.eiice. 
This certainly accounts, at leas-t partially, for thg wide spread investigation* 
pf studentt ratings. A few hours spent in a perusal of publication abstracts . 
from the past fifteen years or so will, reveal to the reviewer the upsurge in 
interest: with regard to the topic. Though there isf evidence to support the 
idea of at least a temporary decline in the use'of student ratings to evaluate 
faculty (Gustad, .1961, 1967) the overall .trend seems to .foreshadow a trend 
of, increased use of student ratings for all three purpose categories. Tlic " 
growing scarcity of financial resources for education, the concurrent piossurti 
,for accountability and similar phenomena are no doubt contributing to this 
increase. Strong' advocacy £oJ* the gathering and use of student ratings 
has also emerged from the students themselves (Werdell, 1967), and even Xrom 
faculty (AAUP, 1974). 

Though often greeted with initial hostility by many faculty members, . 
especially when the formulation strategy of the device did not include their 
input or where the purposes to which the data will be put arc unclear, or arc 
viewed as unacceptable, student ratings will most likely he a part of life in 
higher education for some considerable time to come. In the majority oC in- 
stances, faculty response probably will follow to a large degree, the pattern 
of initial hostility, gradjial acceptance, and eventual support, found by 
Slobin and Nichols (1969). Costin, Greenough, and Monges (1971) provide 
a concise sun^ary of the arg^umenrts, set forth by both the supporters and 
opponents of student ratings, particularly with respect to their proposed 
role in faculty related decisions by the administration. 



PROPOSED ftPPRQACHES \ ' ' 

tr * ^ 

* ft , . - 

The approaches used to cpllect student »:eactions .to instructional ex- • 
periences, have taken widely divergent paths and the important relationship 
between the purpose to which the data has t^ be put^ and the apprpach used , 
^ in developing the instrument has not always been clearly thought out* ^k)SJt 
instruments have'been the result of student groups, individual faculty members 
or departmental committees. On f *r fewer occasions have the devices been 
the result of concerted work by specialists in educational measurement, 
Overv;helmingly they have been predominately teacher-oriented and havp tactically 
^assumed a very' narrow view of the instructional process. It is reported 
that some of the earliest American studies of student, ratings,^ done in the 
1920 's by Edwin Guthrie, were accomp,lishcH]_h3^_gLgki ng__s^^^^^^ ts_ ia 'a-j^X^a4 gkfe — — 



forward manner to describe what* they considerScI to be the salient qualities 
of good teaching. Content analysis was then employed to reduce this raw 
data tQ a format useable for presentation to subsequent groups of students 
who indicated which chai-act eristics were applicaBle to the best teacfier with . 
whom each had studied (Frcnch-Laicovik, 1.974), Evoutually^ a spalo was developed 
from this early work which proved to'* be tl)e central technique of measurement 
used in Guthrie's later research on student ratings ' (Guthrie, 1954), A 
variaat to this strategy utilizing critical incidents has also been attempted 
(Owen, 1967), While this general strategy has been extremely influen tial 
as judged by its frequent use in other investigations (Smith, 1944, Crawford 
and Bradshaw, 1968), it has received criticism on the grounds that it provides 
insufficient guar^antees that all relevant variables hnve been included and 
that it fails ^to indicate the relative importance of each characteristic 
perceived as being involved in good teaching (French-Lazovik, 1974)* 



The notion of relying on ftudenq self-reflections in order to discover 
.what they supposedly value in teaching was avoided by Coffman (1954) v.'heiv 
he simply factor analyzed* a student rating device in use with an aim toward 
.revealing the central factors which had resulted in the student ratings- 
obtained/ Whiie such an approach most g^ertainly spoke to the inadequacies ^ 
ascribed to the Guthrie strategy, the particular device analyzed contained 
relatively £ew items and was rather restricted in th^ scope of the instruc- 
tionally related topics covered » Ad:jiitionally, French-Lazovik (197^^ 
suggests that the application of a prediction model would have proven wuch 
more useful since it woul4>havc yielded information cibout how well students 
e^imates of teaching effectiveness could in fact be prcdict^ed as well as 
revealing the best predictors and their relative impact.- < - . y 

A number of investigators have attempted through factor analytic' trech- 
niques to survey a wide ^range of instruments and items in order to*retrieve 
the essence of student ratings of instruction* In one of therfltigst such 
e'^f forts McKoachie ct al, (1964) reviewed a large number^ of devices \-7lUch 
had been used to collect student ratings of courses and instructors and tbrou^ 
factor analysis attamptecl to. find factors that would be consistent across 
time as well as across ""different students and different teachers. Kaiser's 
method of studying factor similarities yielded six factors meeting sucb 
criteria. Skilf, overload (relating to student work), structure, feedback, 
group interaction, and student teacher rapport were the labels assigned to- ^ 
the resultant constructs* Similarly Hildebrand, Wilson, and Dienst (1971) 
used a large item pool and through ^iscrim^inant analysis sought a few factors 
which would prove to be the central contributors to the wide divergence of 
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student ratixigs. However methodological problems plague such approaches 

•*to the problem, whether orcnnot a prediction model is used as the focus of . * . { 

the study; For example, large numbers, of varl^bites are involved,.- atid t,here v. 

Is ^substantial disagreameut among knowledgeable profassionals not only ' , ' 

to chc ideal slze'and composition pE the item ^i>ools to be ^ixamirted but wi^bh 

regard to the most suitable techniques fqp subsecfuent data analysis as well. 

Having .examined some of th<? methodological "pluses" and "minuses" of 

each investigation, and, subsequently searching "for commonalities among a 

■ number of. factor analytic studies which attempt to account for extent variance ^ ' 

in student ratings, Frehch-La^ovik (1974) concludes that stiidonts seem to 

rate hiahlv instructors who tend to be clear In setting -forth what is to' be 

" ' f -- "* « 
^. ■■ • . ■ 
learned, who in some unspccilied manner bring to life or broaden student ^ 



interests,' and wlto biing. about Increased intellectual activity and higher 

>•,''" 

general motivation in studen^. It wJxild seem that such results could bo . 

• achieved -by/differing -instructional approaches, and by faculty members wljo 

differ Ih style or. personality variables.*' 
- .(t . ' , , 

Silberman and Ailendor (1974) assert that a more helpful course of action • 

' ■« 
at least from the standpoint of providing feedback to instructors would be 

t^'use a semi-projective techniquo. Tlieir device, consists of a written 

response from students of some fifteen minutes duration based on the liypotheticgl 

context of telling a prospective student about the course. No constraints 

or guidelines are. given to influencS the direction or content areas of the 

student responses and content analysis procedures are used to quantify the 

results wliich are divided into scores for ^'evaluative tone" and "input". 
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This latter score relates, to a {i^ersonal growth dimension vJfiich has so often 
been seen as being a very dosixablc but elusive variable on which .to gain 

solid evide?icej;.^2frhe desire of institutions to sbow some positive change ^ 

• . ' ' ^ • ^, 1 ^ . 

in student personal growth as a result of orjjanizcd educational experiences 

appear^ in a device/ devel^ed by Hartley and llogau (1972) in which this 

variable is the concern of half the items. There wojuld seem to be no 

appreciable opposition to gaining such evidence from student ratings il; it 

^inr fact eKists, and this instrument is notable for 5uch an attempt, 

(■ 

Yet another organisational pattern is exemplified by a- device desijjned 

* ft _ ~ ' 

to fit a wide variety ^of course offerings and or^ani5?od on what would" spem 
to be a "logical elements" approach. ^/Ilxc l^^AKE (Sotkloff and Deubidr, 1S^71) 
c-ontains Items aimed at tapping six areas oi course related stimuli^ The 
Professor; Text and Readings j Rxaiftinatioas ; Papers and Reports; Recitation; 
and laboratory. Not ^oaJy can the instructor aeiect areas of the instrument 
'\Aich ^are liiost applicable to a pa^tlcuJar coursc,^l)Ut the PACK is noteworthy . 
in its attempt to move av;ay from an almost exclusive emphasis on instructor 
characteristics and behavior found in tlic va^t majority of devices used to 
solicit student ratings. - * 

Concomitant with the increasing interest in and use of student ratings 
has been an iricrea=yejcl interest on^thc part of commercial publishers in^ the 
development and distribution of instruments designed to be both usab*le and 
useful in a wide variety (^f institutions and instructional qontexts. Perhaps 
among the best known and most widely used iS the Studimt Instructional Report 
(SIR) (Centra, 1972). Shaped partially by facutly review of proposed iteiTjs 
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as.wVl as upon instructional- "factors" found in nrcvioas research, this 
popular' instr'ument^saekS studon? responses" along a ^ur'point scale ranging ; 
from "Strongly Agr(iG"-i;o "Strongly Disagree'; with an additional category of. j 
"Not Applicable or Don't Know?'. A few items are' formatted on a -slightly ' ■ 
different scale. Content areas; include Course Organization Jand Content, . .' ■ 
Instrucltor-Student Relations , 'communications , Assignments and Evaluation, 
lecture. Discus^ion-Seminars, Laboratories, and Student Involvement, -in 
addition to respondent background data. ^ A co™.ent.s section is also i^luded . 
«hiih. encourages open-endQd written responses on any of the above topi'cs, " 
and an opportunity is also pro'vided for '^tu^ent critique of the device itsfu;. 
Advantages of this insttument are its wide applicabilit^and its focus on 
a clear purpose: feedback to faculty for course improvement. Yet there < 
are inherent limitations In a device with "something for everyone" ^and it is 
likewise not uncontn^on for con«idorations related to projected commercxal 
success and wide niarket appUcabili:ty to overshadow such concerns as theo- ^ 
tetical clarity and the uniqueness of particular 'contexes of use. In addition, 
\'t is not clear .whether the use of faculty to screen items and remove those 
of which they did not approv^, contributed t^ strength or weakness ih the " . 
. device. • * ' • 

UNDERLYING GOMClilRNS. bP- liVALUATION . ^ 

Reg^dless. of advantages and disadvantages connected with the particular^ 



approach and format of data collection eCforts or the philo^ophTcl viewpoint S . 
. represented in thei.r .'node of development .nd construction, there are underlying 
: concerns in conne^Jion with virtually every student evaluation that remain. the same. 



Among fhese arc the stability or reliability oF ratings and their validity. ^ 

Correlations b<2tween ratings made at intervals hanging f rom, two weeks* (Lo veil 

and Haner, 1955) and half a semester (Costin 1968) to a year (Guthrie, 1954X 

have consistently produced correlations of .70 and higher. Several different 

. f * ■ 

instruments, have also 'been found to have considerable internal consxstency 

(Spencer and Aleamoni, 197(1; 'Harvey and Barker/ 1970; Guthr.ie, 1954; Maslow 
and Zimmerman, 1956). Regardless of the particular method utilized to calculate 
reliijility, there is considerable cause' for stating that students can irate 
instruction with substantial consistency. The issue of validity. However, ' . 
raises a-number'of dif fi-ciilties v^ith regard to-whut evidence is^accepted as 
being 'sufficient. McKoachie (1969) has observed that faculty members could 
*judge the .validity of student, ratings by matching thom against«thc goals the 
instructor held for the course. ; l/v what has been termed a "student >s con- 
sumed' orientation, evidence exists that students tend to give higher in- 
structor ratings in relation to the extent that they perceive they have met 
coucst* objectives (Lathrop, . 1968) . This way of viewing validity would also 
apply to further course. enrollment by studcuLs who have given high ratings.. 
McKoachie and Solomon (1958) report' some evidence of such a "relationship 
though the evidence is not always consistent (Tobias and llanlon, 1974). 
An alternative source of evidence regarding the validity of student 

* ' • * 

ratings is found in the ratings given instructors or particular courses by 
supervisory personnel or other colleagues. Tfhough significant correlations . 
• betwe(}<fc#student and supervisory ratings are not always found (Webb and Nolan, . 
1955), most, positive results have btsn suiCiciently low i.n magnitude to suggest 



thiit student ratings ar.e. substantially difbrent ^in nature and have thcit own 
contribution to- maka iri attempts to evaluate' instructional experiences 
(Costin, Greenough.'Mejiges, 1971). It might be argued of course that the 
variable of experience and training is the variable to be oxamined^rathor ^ • ^ 
than-stiporvisory or cblleague ratings. Findings from such' studios, however, 
range from negative" correlations ' (Raydcr , 'i968) to no significant rdlationshi-p 
(Heilman and i^mentrout, 1939), to a positive correlation between experience 
.and Ratings .^Walker, 1969) and between academic j^ank and ratings (C^ge, 1961);.. 
The /indings of these and. other similar studies can however -be -undqrWd^ 
'more clearly a^an analysis is made of the particular definitions oC "experience" 
and Straining" which are used. Substantial positive rolatioriships/ can be 
shown between spe'cific aspects ol: student ratings and. precisely defined ^ N 

measures, of training or experience. ^Costin (1968), for; example, found that , 
student ratings of graduate assistants on subiopics as-jlfe^lb-dck" and "group • 
interaction" were clearly related to \<^iether or-^iot the 'assistant^ had com- 
pleted a course emphasizing iristructiorial .techniques applied to the suo^oct . 
area in which they ware teaching. . » » -r^ . 

Perhaps the most • intensely- examined source of potential validity Jor 
student, ratings Ifas been the realm of performaneo. llio-notion is that^if 
student ratings of the instructional experience arc .quite posi-tive then it 

.is reasonable to expect that a. greatey quality or quaivtil:y of learning, or ' 

■• ' . J . i. 

.a. better academic pi/oduct will result. The active debate over some studies 

of this nature whidh have been undertaken is no doubt related in part to the 

^technicaL difficulties which accompaV.y the„. execution of such studies ^Cronbach 

-and Furby, 1970). . For example, (l^din and Rodin, 1972) reported a uegatiVo 



relationship between achievpjgiant and 'student ratings \>\xt the study^ha^; been* 

widely criticizect. Me thado logical 'flaws may well account for the mature of 

<i • , \ • I 

the findings (Rodin, ^'rey, and GassneiT, 1975)* , As Eagle ( 1975)>- points out , 

not only have subsequent, studies failed ^to replicate the findings, but^in- 

vestig5ft0rs s'uch as* Bryson ^1974), Frey (1974)>,^nd Gessner -{1973) haVe all . 

tfound ac^iicvement-rating correlatioi% to be positive in» nature. -It -should , 

be clearly noted, hov;ever, thai- finding a linl<» between p,erformance or achiovo- 

ment and student ratings is quite different from finding such a, link between 

ratings and the biasing variable of grades. Relationships of th<?, latter ^ 

sort J have been v/eak in magnitudu (Costin, Grcer*augh, and Mengcs, 19^1). • 

In a very well-executed attempt to deLermiac if the positive. relationships 

between achievement and ratings found in earlier studies *(Frey, l97"i ; ' 

• ♦ 
Gussne*r,* 1973; Sullivan and Skanes,. 1974) were reliabloy Frey, . Leonard, and 

Boatty (1975) procjuce uvicjcuce which ancRcatos that .stable links can be 

^established between a number of aspects -found in studuit ratings* and sub- 

sequent measured achievement* This "directional thrust^ as 'Scott (1975) 

phrases it,' is thcv most significant facet of the stucfy* Since one barVier ' 

* , ^ * ^ , * , 

to obtaining- strong, consistent ^Alicjlating evidence for student ratings has 

been the inadequacy of the instruments used to measure student learning^ Scott 

urges further* effort toward criterion meaburcs which v;uuld allow^ a more com-' 

plete examinatioit of the rt^^^ Jo^^^ll^ip between achievement and ratings, *One* 

I i " " % , 

possibility suggested in ^tffis * regard is tW^use of matrix item sampling 

(Sirotnlk, 1974), / ^ . , ' o • • 

. '\ ' % ■ ' . 

^» . ' ' ' 
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The central issue in the quest for the establishipent of validity for 

••^student ratings is that highlighted by Crittendon and "Nor r (1975) • Ratings 
must be distinquished from variables which act as 'TDia^ing factors". Class 
size, student sex, major, Gthnic group, and a host of other influences grust 
be found to be substantially independent of ratings. Useful references to a 
myriad of- such investigations are readily available (Crittendon and Norr, 1975; 
Mehges, 1572;^ Centra, 1973; Costin, Greenough , -and Menges, 1971). When this 

'-'^^mafirsive amount of data* is surveyed it is in some ways remarkable and as <, 
>Bagle (1975) notes "encouraging" that the ^porrelations are for the most part 
relatively low. These so-called "biasing factors" simply do not account for 
the majority of .the variance found in student ratings. It, would seem that 
information about otudent percepti^ons of instructional experiences is suf- 

^ ficiently independent from other sources^of evaluative information to be one 
of a number of useful "windows" (Scott, 1975)> through which to view instruction 
Similarily it appears that* the information obtained can be both reliable ^ud 
valid, and that such data can have a useEuj impact on students, faculty, and 
administrators (Centra, L973) . Each .instrument must, however, overcome the 
challenges of issues related to validity and reliability fo^.no blanket 

, validity exists (Kagle, 1975), " 

■ .' \ ' ' ' . • ' 

I^TIONALE FOR ADDITIONAL INSTRUMENTATIOV 

Given the abundance of a wide, variety of student rating forms (Werdell, 
* 1967; Ebefj^^>1970 provide a survey of many currently in use) it seemb. on the 
face of th:^ngs to be an ill advised task to formulate, yet another instrument. 
Perhaps the key to understanding the initiation of work, on instrumentation 
'related to stu^eht ratings at College of the' Mainland lies in a suggestion 
by Halstead (1970) thaf rating devices should be geared to the specific theo- . 
retical podel of instrjuctijOn being applied in the cburses to be 'evaluated* 
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This Suggestion l^as, it appears, gone unnoticed. Current instruments seem 

" - '. - .™ ... _ ^ 

to assume that instruction consists of a series of lectures which accompany 

limited discussion, and/or a laboratory experience of some form* Such de- , 

vices are almost exclusively formulated on the basis ot teacher behavior or 

ft 

:^nferred traits and reflect a4iighly instructor-dependent model o£ learning. 
Additiomiily, the possibility of providing computer generated results, in- 
eluding comparisons with other horizontal or vertical categories of instruction 
within or outside an institution, as well as indicating in a like manner 
possibly* sources of assistance for faculty revision of instructional expe- 
rience>;>, has scarcely been acknowledged* IXie to these and similar reaso.ns, 
ahd 'as the outgrowth of an institutional commitment to programmatic solid- 
tatrioti and utilization of student opinion, tJie present project was undbrtakqn. 

DEVELOPMENTAL GUIDELINES ' ' , ■ ' ^ 

■ -■ ... - ■< — • ^ • 

, la harmony with general systems thinking as applied to educational con- 
texts, an examination of the current state of collecting and using student 
reactions to instruction was conducted* Thds effort yielded ?the khLwlodge, 4 
that while there were a number of quasi- formal student reaction devices in ^ 
use, they were of highly uneven quality, usu^^lly containing 'serious flaws 
from a measurement point of view and often, the kind of information that • 
was obtained eould as easily have been gained from a supplemental section 
added by the instructor to another device. There were however at least two 
exceptions to this goneral state of affairs and these two instruments will . 
prove 'useful in latter s?tages of developing an institution* wide program of 
"assessing student perceptions of instruction/ One of these, devices, ^ur rent ly 
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being used with entering students suffering from severe deficits in terms 
of basic skills, might well become an alternaCe form of the instrument described 
hereip. , On a college wide level, the form most widely used underwent the 
scrutiny of both internal and external evaluiators. The conclusions at which 
both individuals arrived were in essential agreement (Dobbins, 1975), and 
reflected the arL,as of. inadequacy outlined above together with others of a , * 
more technical nature. To further shape/felic direction of the instrumentation 

effort, several administrative leaders within tlie institution who were dircc- 

" ♦ - , 

tly concier.ned with instruction provided a number of constraints upon the do- 

vice and a series of desirable specifications to* be mec. In summary the 

expressed desire was that the ^l^^^i product h€* - . ; ' ' 

V . ^''based in and' refleative of the institutional commitment • to 

what would be termed systematic instruction. This model* 
-"of instruction, adopted and fostered by the college, in- ' , 
^ volves the use of concepts auch as student pre^ssessment., 
^ clear objectives across all categories and levels of the 

,cognitivo and affective d<imains, objectives tied closely 
to behavior, flexibility and ready availabi'lity of learning 
materials, continious t5rogres*s and variability in instructional 
pace, and non-punitive grading directly related to course 
» objectives ' ^ / ^ . v.- 

. * applicable t^o- all divisions within jthe college including 
aca^demic, vocational-technical and non-credit offerings 

appropriate .for administration in cjuasi- traditional courses " . 

and ^should provide some indication of the degree to which 

a course is moving toward the model of instruction, .encouraged 

by the college * * - • ' 

? 

^ k * 

* feasibly administered at any time during the duration of 
a course. (Many courses at the college involve continuous 
registration and course length varies widely as more 
dimensions if instructional flexibility are. introdu'ced 

administered within 30 minutes" or less and involve a minimum 
*^ • amount of faculty time rclat'ive to admini'stration 
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scored by computer and returned to the faculty in time 
for results to influence the subsequent offering of a 

course " , - 

<• ^ ' 

amenable to faculty self interpretations with a mini- 
mum reliance and assistance from outside the Institution n 

* 

flexible enough to allow for collection of additional * 
information by individual faculty" 

designed so as to provide comparisons between instructors, 
sections of given course, course .groupings within a 
3^ division (departments), and divisions within the collegq, 

as well as comparxSOTS" between a particular course section 
and each of these instructional levels. 

capable of mc^asuring^ student preceptions of personal 
growth and an emerging "independent leartier" orientation 
l.\<iiicrr has been a high^.priority outcome sought by the 
- ^ college sxnce its* inception, , ' 

SELECTION OF -FACTORS AND ITEMS FOR FIKLD TEST • • 

Provided with these product specifications and soma notion of what has 
* r * 

teen donO institutionally wit.h respect to student p^erceptions of instruction, 

deyelbpwental work was begun within the resource constraints ^allowod in terms 

of time, personnel, \and ^finances, Factors which wero ^.elected as the result 

o£\ an examination of literature, related to" individualized instruction and 

other "systematic approaches, l^aese "factors** included 1) the organization 

and stfucture of the learning eKperiencc, 2) the learning materials that: ' 

are utilized, 3) the perceived value of the course content, 4) the instructor 

or learning manager as* a teach<ir,^ and 5) the quality of the interactions wliich 

take place betweei^ the students and the instructor or learning manager. Many 

of the' items chosen to represent these factors as the initial version of tb^ 

instrument were those found to have been highly associated with constraints 
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of a like nature in research with othe;r student* rating devices. Preliminary 

* » 

statistical studies show these items to be reasonably stable. Others of course, 
' represent an attempt to find additional items which will serve to insure a 
statistically acceptable minimum number of items for each factor. 

A nvimbcr of psychological advantages to the individual have been claimed 
to accrue to students exposed to flexible systematic approaches to i-nstructioa 
(Young and Van Mondfrans, 1972). Due to sucK claims and an institutional 
stress upon promoting student growth through instruction, a number of experi- 
mental items were included which relate to d hypothesi?:ed factor of "perceived 
personal growth. *\ This factof . is Q,t present a hypothetical notion and evidence 
may or may not be found to support its presence. It may in -fact be related 
to the dimensipjf of student involvement or. personal needs as found in some 
previous studies or highly similar to a scale of internal-'^Kternal locus of 
control. Additionally, two items which solicit information of importance to 
a wide variety of personnel at'the instituLioa arc present. These relate to - 

* a perceived incidence of cheating and a student analysis of the evaluation 

♦ 

activity itself. The latter is considered of high priority in College of 

the Mainland documents relating fco systematic course development procedures 

to be used at the institutiuu. . 

The primitive edition the instrument now beinj^ field tested for ^sub- 
^ V ^ • 

sequent editing and revision albo contains an abnormally large number of items 
, related to student background characteristics. While some'of this information 

will yield helpful additions Lo the currently held profile of the college student 
' bpdy, especially with regard to those enrolled in non-credit offerings, most 
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of these will be useful in confirming the relationship or lack thereof between 
ratings and numerous biasing variables and will be made use of in subsequent 

validity studies connected With the instruments refinement. Since its avowed 

; \ 

purpose is to *tap student perception^ of where a particular course stands as 
currently constituted with respect to certain dimensions thought to be an 
integral part of the model of instruction, adopted by the college itrlTas'bean 
christened, Student Assessment of Systematic Instruction (SASI) • 

SASI: CURRENT CONTliJ^T AND FORMAT ' , ^ « , \ ' 

/Tlie specific items grouped according to the factors which constitute 

the field test version of SASI arq^ listed 1) clow • /it should he'noted that as 

they appear on the student rating form itself, approximately half tHe. items - 

associated with, each factor are reversed in direction to avoid the effects 

of response set, or inattention and all factors* arc' systematically rotated 

with* no two consecutive items representing .the same faqtor. 

, .1^ ORGANIZATION & STRUCTllRlii OF^LriARNINO ' , 

» 1. Wh&t I am to learn is made clear. i ^ 

2\ Assignments arc clear and dofiaite/ 

3.; I understand how I am being graded.- 

^ 4. I am graded* on the .basiis of what J am expected to learni 

5. Classes meet and are dismissed on timc» 

6. The amount of work assigned i« about right. 

J. The time allowed to cover the material is about right. 

^' 8. I received a printed course document which is useful* \ 

9. The course is well organized. 

10. The course is following the printed course .document . 
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II. LEARNING MATERIALS 



1. - Printed materials are readable. 

2. Th^ textbook is satisfactory. . 

3. I like the materials used. . , r i 

4. The. audio^visuai materials fbr this course are helpful. 

5. Students have a chpice of learning materials. 
6*. Necessary materials aire -usually available. 

7. The learning materials aire helpful. 

8. Materials are uhderstandable. 

9. The library is helpful to me' in this course.- . - 
10. Materials seeirt to be presented on my level. 

^- IIL... PmCEIVED VALUE OF CONTENT 

1. What I am learning is worthwhile. 

'2 Important topics , are stressed. . ; «. 

1: I Save become interested in further study of the subject. 

4 I can use what I am iearnfng, - ' 

5 I would- rate this bourse higher tl^an others I have takon her 
e! Taking this course has been a gobd^exp^eriance. 

?•. I attend this class regularly. - ' '\'- ^ 

8. I would rate, this course higher than most I have taken. 

9. Most people would benefit from j|bhis -course. 



> i 

IV. STUDfiNT PERCEPTION OF PERSONAL GRdWTH 



1. This coutse is. "changing my vj.ew of myself. _ ^ 

2 I am developing a better understanding of other people.^ 

3 This course is helping me rely more on myself. 

4. I feel closer to olther people" as a result" of thxs course. 

5. 'I now know more about myself. . 

6. ' I will >,npw,ij?:ob ably, study more on my o^n. 
7 This course \s changing how I think. 

8* "it is easier now to see other points of view. 
.9! I am learning that ! can change the way things are. 

V. CLASS LEADER OR MANAGER AS A TEACHER 

^ . 

1 Oveirally the instructor is a good teacher. - 

2. The instructor seems to be interested ih> the subject. 

3. The instructor speaks clearly. 

4*. Tlie instructor answers questions. , . , . . 

5: The instructor knows a great^ deal a-bout the subject. 

6* The instructor is sure and confident. 

7'; The instructor is usually, free from annoying manneri'sms. 

8. Th^ instructor °.ps -my interest.. 4„^*.r-urtnr 

9 I would like to ke another course, from this instructor. 

10' The instructor's answers to questions are accurate. 

Ll! The instructor can handle difficult situations. 
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VI. QUALITY OF STUDENT INTERACTiON 
l«TH LEARNING MANAGER OR TEACHER 



1. The instructor keeps regular office hours. ' 

2. I can get help from the instructor outside of class. 

3. The instructor shows respect toward students. 

4. The instructor takes an interest in students, as persons. 
5^ Students can express their opinion. 

6. Differences of opinion arc allowed. 

7. Grading is impartial. 

8* There is quite a bit of student participation in the class. 

9. ^The instructor seems to lijce being with the students. 

10. No one in this .course gets special'treatment. 



• ' VII. INDEPENDENT ITEMS ~ — 

1. There is no cheating*. . * ^ - 

" 2. .Having students evaluate courses is wortliwhilc. ^ 
3'.^.jSupplementary questions prepared by instructors. 

In terms of individual" data, the student- is asked to indiCcite : 

• 1.' :^pe of progran^ in which ho^iSL^nroi led. 

'2. Current enrbllment^^pattern with respect^to credit and non-credit 

. courses. ^ x - ' ^^^^^ - . 

3. Number of semesters of prior** attendance at the institution^-^, 

4. Extent and type of current employment. . 

5. CharacteristiiC level.'of perceived ^academic* per for,mance. 

6. Ethnic group. 

7. Tfie number of pre-school children .at 'homo.. . ^ 

8. Birthdate. ' , . . ' 

9. Sex. ^ ^ 

10. Previous number of times this semester the SASI has been filled out*, 

11. Language commonly spoken at home. 

Wira RESPECT TO EACH ITEM, THE STUDENT IS ASKED TO RICAD EACH STATEMENT 
CAREFULLY AND DARKEN THE SPACE ON THE ANSWER SHEET THAT BEST DESCRIBES 
HIS OR HER OPINION ACCORDING TO TllE FOLLOWING SCALE: 

(1) Strongly Agree , ^ou strotigly agree with the stat^ement as it 
applies to this course. . * ^ . - 

(2) -^^^ agree more than you disagree with the statement as 
it applies to this course. 

(3) Disagree . You disagrce*more than you agree witli* the statement 
as if^applies to this course. 

(4) S trongly Disagree . You strongly disagree with the statement as 
-•it applies to this course. 

(5) Not Applicable or Don^t Know , llie statement does not apply to 
this course, or you simply are not able to' give a knowledgeable 
response* 



STEPS OF DATA ANALYSIS 

Two levels of analysis are performed for each of the item groupings in- * 
eluded in the instrument. /Ilie first level of analysis is designed to answer 
the question ''With respect to each factor measured, how does this section as 
a wbole compare in relation to other sections of the same course, the-,sanie *~ 
department, the same flivtsion, and in the college?" llie answer to the questio 
is given through use of one, of the following; terms: high, above average, 
average, below average, or low. The computation proc€idure utili?:ed to execute 
the first level of analysis can be reduced to the following steps: ^ 

1. Determine the common variance of the items in 
an item 'group, ' 

2, % Establish weighting for each of the items in the 

item groupr^.based on the degree of common variance 
of the item with theP'other items (some items v^re 
1 better measures and therefore will carry higher 

weight). ^' 

^ ; ^3, '.Compute* the group statistics .for the item group 
mean (or average), Standard deviation, standard 
error :of the mean — computations performed by,. ^ ^ 
class, course, department, division, and college. 

^ 4. Since the' aritlimetlc mean -(or 'average) is subject 
to* errors of measurement, calculate the confidence 
interval^ about the mean sucii that we can state with - < 
95% confidence that the "true** mean lies within the > 
'\ ' interval. 

5. Determine the item group rating according to the 
• — - following rules: A ^ " . 



a. Haviitg oiStablished the confidence interval for a 
x given class, rate the class below averd'ge compared 
' \to college,- the course, department, division, if * 
^ the mean pf the designated groupdng f:alls within 
• the confidence interval of the class; and rate the 
class ahove average if the mean of the designated . 
* /grouping rails below the lower limit of the class 
confidence erval. " . ^ » 
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b. If the class is rated above average -and, the class 
mean is Itf*^ or more points above the^ mearir^of the 
designated gtoup, revise the rating of ttie class> 
to "high;" or if the class is rated below average 
* , and the class mean is 10* or more points, below 

the mean of the designated group, revise the rating 

given the class to ''low." ^ 

» 

*Note: All scores are converted to standard scores 
# , with a mean of 50 and a standard deviation of 10 

for statistical reasons. Hence, 10 points Is 1 
standard deviation unit. , . 

The second level of analysis performed for each of the 'item groupings 
designed to answer the question — "how^ may the -rating qbtained in the 
previous analysis be explained?" Our analytical methods can give only a-^ 
partifcal answer, or perhaps only some clues. The answer(s) aire framed in- 
terns of item ratings given in the factor by factor report -Eor eaph course- 
section. 

' The items which form the basis of the 'group-score and rating arc, 
themselvfes ranked high to low .based oa.the mean studciit ratings received 
by each. Tlie weighting ol each^icem is displayed along with the arithmetic 
average and standard deviation of the ratings received. Items receiving 
high mea,n ratings by-^ students and those possessing high weightings ^ith 
regarded/ a particular Jactor are most influential in producing high group 
scores-on that factor. Items having ^loW mean ratings, yet possessing high 
factor weights, contribute to low group scores. Lower weighted items, irre- 

, - : ^ • 

spective qf mean ratings, contribute less to the group score. . 

■ REPORTING OF- RESULTS . * , ■ . 

A report in" the form of a multi-page computer printouts, is subsequently 
prepared for each faculty, member whtch provides three basic kinds pf infor- 
mation. First; student- perceptions relative to each factor in every cdurse 
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section for .which the instructor is responsible ate summarized and Reported. 

• ' » 
Second, a number of relative comparisons are made between student perceptions 

in a particular class with perceptions of other section of the course, the 

division and the institution. Third, an item by item breakdown is given* 

with respect to student responses on every item associated with each factor. 
* ' " • • 

Those items which do. not contribute significantly to* any of the six distinct 
factors currently being measured are grouped together at the end of the re- 
port under the heading of "Independent Item's" with only item information 

being reported since no known factor is involved, * 

^' ■* . • 

Appendix A contains a page from' a computer report typical of that re« 

ceived by each faculty member^. Note that in the top Ifeft portion of the page 

man;J^ of the symbols used in Mhe report are explained. To the right of that 

' - ,^>^-§^ . _ « • , 

explanation is found a table which summarizes the number of respondents to 

♦ - i» ^ 

the instrument in terms of classes, courses, students, and the like. 



-The remainder of t^jp^^intout is presented oft a course section by 
course section basis T;^ith the results (by factor) for ^each. Course section 
being reported in the follo\5^ing order; (1)*' Structure and Organization of 

Learning, (2) Learning Materials, (3^ Value of C9urse Content* (4) Perceiv^^d 

• * * i. * 

Personal Growth, (5) Instructor or Learning Manager as a Teacher, and (6) 

^ «i s , ^ • \ , • 

Quality of Student-lTistructor Interaction^ 

With regard to each factor, results are presented in a^fi identical 
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sequence. On thjB^ left, the class, course, department, division,, and college 

* —~ ^ " ^ 

mean ratings for the factor* in question are listqd along with numeric limits 
within which we can be confident (at a 95% level) that the "true" mean rating 
lies. This concept of confidence limits, though perhaps initially difficult 

23 ^ 
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to grasp, is an important one. All measurements contain some error due to 

imperfect instrumentjatiorij response variables, and a plethora of other in-* 

fluences. Even repeated measurements with a refined instrument under highly 

* , •* 

contrdlled conditions would not celiminat(j(; all^ sources of error. Any^ scores 

or measurements we present then only approximate a true and ;|ccuratc measure- 

ment. Therefore, an attempt has been made to report a range within which wc 

♦ * 
can be confident that the "true" scpre is located/ Additionally, the standard 

deviation, which is an indication of. how dispersed or spread^ out the ratings 

are, is provided. In this example, note that the ratings of the class .tended^ 

to be more spread out or vary" to a greater ^extent than did student perceptions 

of the same factpr generally throughout the division. ^ . ^ 

? . , , • • » * 

To the right of this data^ *a graphic display of' comparative arithmetic \ 

means and confidence limits is given with these saniQ relationships summarized' 

f 

t 

verbally -bn the right hand side of the''{)agc- Ona can easily scq, fpr exainple, 
that the mean class perdcption on th«, ^ctor sumraarizad in this portion of - 

^ • * • * - * 

the report ia, with respeetr t:o the mean perceptions- iyi-all sections of the 
same course below average but_is above av^cxage wheif compared with perceptions 
of the same factor throughout the department. . - ^ . 

Detailed data with respect .to each item associated with the particular , 
factor being .reported, follows. At the extreme left of this display the - 
"Rate." column is indicative' of 'whether the average student rating, of the 
item is significantly above,* near, or below' average. The items average 
rating on the four point response scale being utilized follows, along with 
a measure of how dispefsed or spread out the ratings were on this item (SD). 



Subsequent columns in this display indicate (1) the relative importance or 
impact of the item in the factors composition', (2) the percentage of 
respondents checking the "not applicable/don't know** option, (3) a classi'V 
fication-of this percentage as being significantly abpVje, c/near,jOr below 
average, and (4) the number of the item as it apprearad= in the instruments 

Items are listed in an order reflecting theit positive contribution to 
the factor i^ati'tag/ Examining item weightings and printed order visually thus 
provides a simple and rapid method for discerning potential reasons for ob- 
tained factcn: ratings. The same data might, be utilir.cd as initial guid,c^- 
posts in determining likely strategies for increasing student ratings on a 

\ . " ' ' ' ' ■ ' ■ 

particular factor. Note that items which were originally |)hrased ±n jieiiPXiyc 
terms have, in the .^ireported results, been Reversed and are stated in the 

positive* ^ . / 

'Item. information includes an analysis of the degree of occurrence 
of missing responses and responses to the "not applicable" .response alter 
native. Overall response, rabes are computdid item" for the college , as a 
wliole and pompari'son is then made to the item response rates for each class. 
Differences are indicated when the number *of misusing and "not applicable" 4 
responses differs significantly in the class. 

Additional information concerning the number of occurrences of "no 
response" and^ "not applicable" responses is provided with the item displajf . 
which* contains the mgan rating ana standard deviation* Items will bp excluded 
from reports* for a particular> class '^att'alysis:^whei\.^ the number of responses 
indicajting "no.t ap^licable/don* t know" exceeds 50% of the total number of 
Students in the class who fill'out the instrument. 



CURRENT . FIELD TjEST ' , ^ . . ^ 

The instxument as herein described was .administered to a non-represent:ative 
sample of the student body ^liring December, 1975. I^istructors administered the 
form on, a voluntary basis but 'all instructors were enQouraged.ito utilized it 

with at least on$5 of their classes. It is/estimated tftat between fen^*ancf<' • - 

' ' " \ , ' . ' 

fifteen pQtrcent of the fall student^enrollanent respdnded at least ohce to 1 

» 

the device but at time* of administration owing to the degree on instructional - 
flexibility alre^y extant in the normal operations of the college, many* 
courses had' already concluded, and a number of additional students had already 
^tjmpleted courses that* were, for other students, still operational. - While 
instructors wer*e encouraged to supplement the 'device with items covering con- 
cerns unique to a particular class or course, most gained such data by using 
whatever devise they had employed i>reviously« Therefore a number of students, 
responded to more than one. device a\n evaluating a particular course* Semi- 
structured or open ende^jf iteiiKadditions were alsb encouraged. Anecdotal 
reports indicated' ah average response time of appro^x^imately '25 minutes for 
initial responders with a response time approaching '15 .jninutes for some 
studenrts vho completed the form in a fourth *6r fifth class » Owing to a 

/ • • h 

number of* variables adversely effecting the fall 'administration, ^ * 

including "^sample composition, the unfortunate timing of niaterials distribution " 

including a delay in placing a user^s manual (Nelson and Dobbina, 1975) in . 

the hands" of faculty,, a decision was made to process inforrgation. from a* far ; . 

more -extensive administration of, the SASI in March ol; 1976, prior t<5* revising 

^* 

' ' . . .* 

and refining the student form and data collection procedures^ 



FUTURE DIRECTIONS * ' ^ 

The time and effort as well as vthe other". resources needed to develop and 



.validate a most roeasurement instrument is substantially indeed. College of 
the- Mainland is committed to using information from students as one of several 

.inputs through which to evaluate and impro -e instructional oxperietices. During 
the cufrgnt academ'ic year a beginning has been made. As one result of the 
stydeht' assessments collected this year, each item will be scrutinized for 
possible retention in its present or an. altered .f,orm or replaced. In a 

"similar vein, the existing factors included in the" field- test version of the 
device will be examined for possibje.chs^nges in number and structure. A number 
of approaches "will be- investigated in a quest for validity. 

With a somewhat more extended time-frame in mind) th'e possibility o.f . 
using computer reports to provide more detailed diagnosis of the aspects of 
Ihstyuction needing modification, and as a referral mochaaism for iiidividual , 
faculty to indicate sources-. of .assistajic in making suth improvements will 
hopefully be explored." .Some consideb£1^^^Vs been giyen to. slvapinp, the 

.current device to a gredter ^gr^ee,' toward ffii^crtoning as a general barometer 
with respect to the major elements of systematic instruction while developing 
a series of* shorter, more focused diagnostic ins trumeBi;^. with wTttch to pinpoint 

' the. specific facets of the instructional system need remediation. ' For example, 

o«e.such diagnostic device might dwell entirely 'on the suitability and aval 1- 

> ' • •• ■ • 

• ability of the learning materials used in the course and contain items related 

to readability, cultural bias,, acco.ssibi^iity and "the like. The overall goal 
" qf such "explorations would "however remain what it has always been; to develop 
. a program^of collecting and using student perceptions of instructional experien 

• • • 

• which is .based on the model of systematic Instruction adopted by College of • 
the ^inland^, , ' On • * 
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REPRESENTATIVE COMPU'£ER PRINTOUT OF RESULTS 
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